d-blink: Distributed End-to-End Bayesian Entity Resolution

Entity resolution (ER; also known as record linkage or de-duplication) is the process of merging noisy databases, often in absence unique identifiers. A major advancement ER methodology has been application Bayesian generative models, which provide a natural framework for inferring latent entities with rigorous quantification uncertainty. Despite these advantages, existing models are severely limited practice, standard inference algorithms scale quadratically number records. While scaling can be managed by fitting model on separate blocks data, such naïve approach may induce significant error posterior. In this article, we propose principled scalable ER, called “distributed linkage” d-blink, jointly performs blocking and without compromising posterior correctness. Our relies several key ideas, including: (i) an auxiliary variable representation that induces partition records into blocks; (ii) method constructing well-balanced based k-d trees; (iii) distributed partially collapsed Gibbs sampler improved mixing; (iv) fast performing updates. Empirical studies six datasets—including case study 2010 Decennial Census—demonstrate scalability effectiveness our approach. Supplementary materials article available online.